Claims about the use of software engineering practices in science: A systematic literature review

Context: Scientists have become increasingly reliant on software in order to perform research that is too time-intensive, expensive, or dangerous to perform physically. Because the results produced by the software drive important decisions, the software must be correct and developed efficiently. Various software engineering practices have been shown to increase correctness and efficiency in the development of traditional software. It is unclear whether these observations will hold in a scientific context. Objective: This paper evaluates claims from software engineers and scientific software developers about 12 different software engineering practices and their use in developing scientific software. Method: We performed a systematic literature review examining claims about how scientists develop software. Of the 189 papers originally identified, 43 are included in the literature review. These 43 papers contain 33 different claims about 12 software engineering practices. Results: The majority of the claims indicated that software engineering practices are useful for scientific software development. Every claim was supported by evidence (i.e. personal experience, interview/survey, or case study) with slightly over half supported by multiple forms of evidence. For those claims supported by only one type of evidence, interviews/surveys were the most common. The claims that received the most support were: “The effectiveness of Preprint submitted to Information and Software Technology June 15, 2015 the testing practices currently used by scientific software developers is limited” and “Version control software is necessary for research groups with more than one developer.” Additionally, many scientific software developers have unconsciously adopted an agile-like development methodology. Conclusion: Use of software engineering practices could increase the correctness of scientific software and the efficiency of its development. While there is still potential for increased use of these practices, scientific software developers have begun to embrace software engineering practices to improve their software. Additionally, software engineering practices still need to be tailored to better fit the needs of scientific software development.